Camera based safety mechanisms for users of head mounted displays

ABSTRACT

Methods and systems for warning a user of a head mounted display (HMD) during game play of a video game. A game is executed causing interactive scenes of the game to be transmitted for rendering on a display portion of the HMD. Coordinates of the HMD are determined in a three-dimensional space of a real-world environment in which the user wearing the HMD is present, to identify a current position of the user. A movement of the HMD is determined during execution of the game by identifying a change in one or more coordinates of the HMD in the three-dimensional space. When it is determined that the user is approaching a boundary of an interaction space in a real-world environment, a warning signal is conveyed to the HMD to indicate proximity of the user to the boundary of the interaction space.

CLAIM OF PRIORITY

This application is a continuation of U.S. application Ser. No.15/832,678, filed on Dec. 5, 2017, and entitled, “Camera Based SafetyMechanisms for users of Head Mounted Displays,” which is a continuationof U.S. application Ser. No. 15/473,469, filed on Mar. 29, 2017 (sinceissued as U.S. Pat. No. 9,873,049 on Jan. 23, 2018), and entitled,“Camera Based Safety Mechanisms for Users of Head Mounted Displays,”which is a continuation of U.S. application Ser. No. 15/088,000, filedon Mar. 31, 2016 (since issued as U.S. Pat. No. 9,908,049 on Mar. 6,2018), and titled, “Camera Based Safety Mechanisms for Users of HeadMounted Displays,” which is a continuation of U.S. application Ser. No.14/333,500, filed on Jul. 16, 2014, and titled, “Camera Based SafetyMechanisms for Users of Head Mounted Displays,” (since issued as U.S.Pat. No. 9,630,105 on Apr. 25, 2017) which is a continuation-in-part ofU.S. application Ser. No. 14/170,470, filed on Jan. 31, 2014, andtitled, “Camera Based Safety Mechanisms for Users of Head MountedDisplays,” (since issued as U.S. Pat. No. 9,729,864 on Aug. 8, 2017)which claims the priority benefit under 35 USC § 119(e), to U.S.provisional application No. 61/884,900, filed Sep. 30, 2013, and titled“Camera Based Safety Mechanisms for Users of Head Mounted Displays.” Theabove-referenced applications are incorporated herein by reference intheir entirety.

TECHNICAL FIELD

This disclosure relates generally to human-computer interfaces, moreparticularly, to warning users when they move outside of a camera'sfield of view or when they approach one or more tangible obstacles.

DESCRIPTION OF RELATED ART

The approaches described in this section could be pursued, but are notnecessarily approaches that have previously been conceived or pursued.Therefore, unless otherwise indicated, it should not be assumed that anyof the approaches described in this section qualify as prior art merelyby virtue of their inclusion in this section.

One of the rapidly growing technologies in the field of human-computerinteraction is various head-mounted or head-coupled displays, which canbe worn on a user head and which have one or two displays in front ofthe one or two user eyes. This type of displays has multiple civilianand commercial applications involving simulation of virtual realityincluding video games, medicine, sport training, entertainmentapplications, and so forth. In the gaming field, these displays can beused, for example, to render three-dimensional (3D) virtual game worlds.The important aspect of these displays is that the user is able tochange a displayable view within the virtual world by turning his headin addition to utilizing a traditional input device such as a keyboardor a trackball.

However, in traditional head-mounted displays, the users are vulnerableto get injuries when they wear the head mounted display and provide useractions, such as making motions or move towards tangible obstacles suchas a coffee table, chair, etc. In addition, in some human-computerinterfaces involving head-mounted displays, the user shall be presentwithin the field of view of a dedicated video camera or depth sensor. Itmay be annoying for the user when the human-computer interaction isinterrupted if the user moves out of the field of view.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described in the Detailed Descriptionbelow. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Generally, the present disclosure refers to the technology for warningthe users of head mounted displays (HMDs) that they approach an edge ofa scene defined by field of view of a camera or that they approach oneor more tangible obstacles. The warning includes presenting audio and/ordisplayable messages to the user, or moving the display(s) of the headmounted displays away from the user's eyes. In addition to providingwarning, the present disclosures may be used to adjust content that isbeing rendered on a display portion of the HMD based on the actions,such as moving toward edge of field view, moving toward a tangibleobstacle, etc., by the users wearing the HMD. In some embodiments,adjusting the content includes rendering real-world object from theimmediate vicinity of the user on a display portion of the HMD. Thedetermination that the user approaches the edge of scene or a tangibleobstacle is made by dynamically tracking motions of the users throughanalysis of images and/or depth data obtained from image sensor(s)and/or depth sensor(s) secured to either the head mounted display,arranged within or outside of the scene and not secured to the headmounted display, or a combination of both.

The edge of the scene may include a tangible obstacle. The imagesfurther include depth information of the scene. The scene can be definedby a field of view of at least one image sensor or at least one depthsensor. The warning includes directing, by the processor, the user backtowards a center of the scene.

In certain embodiments, the method further includes receiving, by theprocessor, secondary images of the scene from one or more secondarysensors arranged outside of the scene and not secured to the headmounted display. The determining that the user approaches towards one ormore tangible obstacles is further based at least on the secondaryimages. In certain embodiments, the one or more tangible obstacles arepresent within the scene. In some embodiments, the one or more tangibleobstacles include an edge of the scene.

According to one embodiment, a method is provided. The method includesexecuting a game. The execution causes interactive game scenes of thegame to be transmitted to a head mounted display for rendering on adisplay portion. The head mounted display (HMD) is configured to be in awireless communication with a client device and to block a view to areal-world environment, when worn by a user. Coordinates of the HMD isdetermined in a three-dimensional space of a real-world environment inwhich the user wearing the HMD is present. The coordinates are used toidentify a current position of the user wearing the HMD. A movement ofthe HMD is monitored during execution of the game. The movement of theHMD is determined by identifying a change in one or more coordinates ofthe HMD in the three-dimensional space. The change in the one or morecoordinates of the HMD is evaluated to determine when the user isapproaching a boundary of an interaction space in a real-worldenvironment. A warning signal is conveyed to the HMD for rendering. Thewarning signal is indicative of a proximity of the user to the boundaryof the interaction space.

In some embodiments, an intensity of the warning signal is adjustedbased on the evaluation of the change of the HMD. The intensity isadjusted to increase as the user continues to approach the boundary ofthe interaction space and to decrease as the user moves away from theboundary of the interaction space toward a center of the interactionspace.

In some embodiments, the warning signal is provided in an audio formator image format or a textual format or a haptic format or anycombinations thereof.

In some embodiments, the warning signal includes instructions to directthe user away from the boundary of the interaction space in thereal-world environment and toward a center of the interaction space.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by limitation inthe figures of the accompanying drawings, in which like referencesindicate similar elements and in which:

FIG. 1 is an example scene suitable for implementation of variousaspects of the present technology, in accordance with an embodiment ofthe disclosure.

FIG. 2 is an example system suitable for employing methods forcautioning a user of a head mounted display, in accordance with anembodiment of the disclosure.

FIG. 3 is another example system suitable for employing methods forcautioning a user of a head mounted display, in accordance with anembodiment of the disclosure.

FIG. 4 is yet another example system suitable for employing methods forcautioning a user of a head mounted display, in accordance with anembodiment of the disclosure.

FIG. 5 is a process flow diagram showing a method for cautioning a userof a head mounted display, in accordance with an embodiment of thedisclosure.

FIG. 6 is a diagrammatic representation of an example machine in theform of a computer system within which a set of instructions for themachine to perform any one or more of the methodologies discussed hereinis executed.

FIG. 7 illustrates different parts of an exemplary head mounted displayworn by a user, in accordance to one embodiment.

FIG. 8 illustrates tracking of a head mounted display and a controllerworn/handled by a user, in one embodiment of the invention.

FIG. 9 illustrates the various modules used in tracking the head mounteddisplay and the data flow between the head mounted display and theclient device/cloud server, in accordance to one embodiment.

FIG. 10 illustrates an exemplary embodiment for determining position ofthe head mounted display and/or the controller worn/handled by a user.

FIGS. 10A-10C illustrate exemplary embodiments for determining change inelevation of the head mounted display worn by the user.

FIGS. 11A-11E illustrate tracking of head mounted display to identify asafety issue, in one embodiment of the invention. FIG. 11F illustratestracking of a head mounted display to identify another safety issue, inan alternate embodiment of the invention.

FIG. 12 is a process flow diagram showing a method for adjusting contentrendered on a head mounted display, in accordance with an alternateembodiment of the invention.

FIG. 13 illustrates overall system architecture of a game module, in oneembodiment of the invention.

FIG. 14 illustrates a block diagram of a game system, in accordance withan embodiment of the invention.

DETAILED DESCRIPTION

The following detailed description includes references to theaccompanying drawings, which form a part of the detailed description.The drawings show illustrations in accordance with example embodiments.These example embodiments, which are also referred to herein as“examples,” are described in enough detail to enable those skilled inthe art to practice the present subject matter. The embodiments can becombined, other embodiments can be utilized, or structural, logical, andelectrical changes can be made without departing from the scope of whatis claimed. The following detailed description is therefore not to betaken in a limiting sense, and the scope is defined by the appendedclaims and their equivalents. In this document, the terms “a” and “an”are used, as is common in patent documents, to include one or more thanone. In this document, the term “or” is used to refer to a nonexclusive“or,” such that “A or B” includes “A but not B,” “B but not A,” and “Aand B,” unless otherwise indicated.

The techniques of the embodiments disclosed herein can be implementedusing a variety of technologies. For example, the methods describedherein are implemented in software executing on a computer system or inhardware utilizing either a combination of microprocessors or otherspecially designed application-specific integrated circuits (ASICs),programmable logic devices, or various combinations thereof. Inparticular, the methods described herein are implemented by a series ofcomputer-executable instructions residing on a storage medium such as adisk drive, or computer-readable medium. It should be noted that methodsdisclosed herein can be implemented by a mobile terminal, cellularphone, smart phone, computer (e.g., a desktop computer, tablet computer,laptop computer), game console, handheld gaming device, and so forth.

In general, the embodiments of the present disclosure provides methodsand systems for warning users of head mounted displays that approach anedge of a scene or move outside a field of view of a camera, or whenthey move towards one or more tangible obstacles. The determination thatthe users approach the field of view edge or the obstacles is made basedon analysis of images and/or depth data. The images and/or depth datacan be obtained from image sensor(s) and/or depth sensor(s) arrangedeither on the head mounted display or apart from the user and not beingconnected to the head mounted display, or a combination of both. Thewarning includes audio messages, displayable or visible messages,highlighting of LEDs, graduate fading of the displayable images on thehead mounted display, fading of the images' brightness, blending in thereal world view, causing the head mounted display to vibrate,disengaging the head mounted display or its parts, or moving display(s)of the head mounted display away from the user's eyes so the user couldsee the real world and obstacles where he moves.

The term “head mounted display,” as used herein, refers to one or moreof the following: a wearable computer having a display, head mountedelectronic device, a head-coupled display, a helmet-mounted display, ahead-mounted computer with a display. The head mounted device, which isworn on a head of a user or which is a part of a helmet, has a smalldisplay optic in front of one (monocular display device) or each eye(binocular display device). The head mounted device has either one ortwo small displays with lenses and semi-transparent mirrors embedded ina helmet, eye-glasses (also known as data glasses) or visor. The displayunits can be miniaturized and may include a Liquid Crystal Display(LCD), Organic Light-Emitting Diode (OLED) display, or the like. Somevendors employ multiple micro-displays to increase total resolution andfield of view. Some other head mounted devices do not use a traditionaldisplay at all and instead project light directly into the user's eyes.

The term “depth sensor,” as used herein, refers to any suitableelectronic device capable of generating depth maps of a 3D space. Someexamples of the depth sensitive device include a depth sensitive camera,stereo 3D camera, depth sensor, video camera configured to processimages to generate depth maps, and so forth. The depth maps can beprocessed by a control device to locate a user present within a 3D spaceand also its body parts including a user head, torso and limbs. The term“image sensor,” as used herein, refers to any suitable electronic devicecapable to generate still or motion images of the 3D scene. Someexamples of image sensors include a video camera, photo camera,charge-coupled device (CCD), and so forth. The term “field of view,” asused herein, refers to the extent of 3D space covered and “viewed” bythe image sensor and/or depth sensor.

The term “control device,” as used herein, refers to any suitablecomputing apparatus or system configured to process data, images, depthmaps, user inputs, and so forth. Some examples of control device includea desktop computer, laptop computer, tablet computer, gaming console,audio system, video system, cellular phone, smart phone, set-top box,television set, and so forth. In certain embodiments, at least someelements of the control device are incorporated into the display device(e.g., in a form of head-wearable computer). The control device can bein a wireless or wired communication with a depth sensor/image sensorand a head mounted display. The control device is also responsible forgenerating images displayable further on the head mounted display (e.g.,generate a virtual reality). In certain embodiments, the term “controldevice” refers to a “computing device,” “processing means” or merely a“processor”.

According to embodiments of the present disclosure, a head mounteddisplay can be worn by a user within a particular 3D space such as aliving room of premises. The user can be present in front of a depthsensor/image sensor which track user's motions. The control deviceprocesses data received from the depth sensor/image sensor and, by theresult of the processing, the control device identifies the user, user'smotions/gestures, and track coordinates of the virtual skeleton withinthe 3D space.

With reference now to the drawings, FIG. 1 is an example scene 100suitable for implementation of various aspects of the presenttechnology, in accordance with an embodiment of the disclosure.

In particular, there is shown a user 105 wearing a head mounted display110. The user 105 is present in a 3D space being in front of a controldevice 115 which includes a depth sensor and/or image sensor so that theuser 105 can be easily tracked. The control device 115 is coupled to oris a part of a gaming device or a gaming console 120, and can be also inoperative communication with the head mounted display 110 and some otherperipheral devices such as a TV display 130, audio system, etc.

In order to be tracked by the control device 115, the user 105 shall bepresent within the field of view of the depth sensor and/or image sensorof the control device 115, which area is also referred to herein as a“scene” 135. If the user 105 moves out of the scene 135, he would not betracked and thus his interaction with the control device 115 isinterrupted. Alternatively, the user 105 can move towards a tangibleobstacle 140 such as a table, chair, toy, and so forth. Because the user105 may not see the real world or have a diminished view of the realworld while wearing the head mounted display 110, he may hit theobstacle 140 and get an injury. Accordingly, the present technology isdirected to warn the user 105 if he approaches an edge of scene 135 orone or more obstacles 140.

FIG. 2 is an example system 200 suitable for employing methods forcautioning a user of a head mounted display, in accordance with anembodiment of the disclosure.

As shown in FIG. 2, the system 200 includes a head mounted display 110and a control device 115. The head mounted display comprises one or moredisplays 250, a communication module 260, and, optionally, a triggeringdevice 270.

In certain embodiments, the display 250 is arranged in front of the oneor two user eyes. The display 250 is configured to simulate virtualreality including video games, sport training, entertainmentapplications, and so forth. In one example embodiment, the display 250is configured to render three-dimensional (3D) virtual game worlds. Theuser is able to change a displayable view within the virtual world byturning his head.

In certain example embodiments, the communication module 260 isconfigured to operatively communicate with the control device 115, andsome other peripheral devices such as a TV display 130, audio system,and so forth.

In certain example embodiments, the triggering device 270 is configuredto trigger a warning signal sent by the control device 115 to the headmounted display. For example, the triggering device 270 can initiate adisplay of a displayable image on the display 250 or the fading ofbrightness of images displayable by the head mounted display to theuser. In one example embodiment, the triggering device 270 stopsdisplaying images by the head mounted device. In another exampleembodiment, the triggering device 270 moves at least one display 250 ofthe head mounted display 110 away from eyes of the user.

In some example embodiments, the control device 115 comprises acommunication module 210, a computing unit 220, a depth sensor 230, andan image sensor 240. The communication module 210 is configured tooperatively communicate with the communication module 260 of the headmounted display 110 and other peripheral devices such as a TV display130, audio system, and so forth.

In certain embodiments, the communication module 210 is configured totransmit images of a scene captured by the depth sensor 230 and/or theimage sensor 240 to the computing unit 220. In certain embodiments, thecomputing unit 220 is configured to process images of the scene receivedfrom the depth sensor 230 and/or the image sensor 240, track thelocation of the user based on the received images and determine that theuser approaches an edge of the scene. The computing unit 220 is furtherconfigured to send warning messages to the head mounted display 110about the proximity of the scene edge.

In certain embodiments, the computing unit 220 sends triggering signalsto the triggering device 270 based on the processed images of the scene.In certain embodiments, the depth sensor 230 captures video data in 3Dunder any ambient light conditions. The sensing range of the depthsensor 230 is adjustable and automatically calibrated based on thephysical environment based on the presence or absence of furniture orother obstacles.

In certain embodiments, the image sensor 240 is a device that convertsan optical image into an electronic signal. The image sensor 240 caninclude, for example, one or more video cameras that are sensitive tothe visual range, IR cameras sensitive to one of the IR bands, UVcameras or other direct or indirect image-generating sensor systems, forexample, radar or laser radar. Images from the depth sensor 230 and theimage sensor 240 can be combined by data fusion and be displayed to theuser.

FIG. 3 is another example system 300 suitable for employing methods forcautioning a user of a head mounted display, in accordance with anembodiment of the disclosure. The system 300 is similar to the system200 shown in FIG. 2, except that a depth sensor 280 and image sensor 290are secured to the head mounted display 110.

FIG. 4 is yet another example system 400 suitable for employing methodsfor cautioning a user of a head mounted display, in accordance with anembodiment of the disclosure. The system 400 is similar to the system200 shown in FIG. 2, except that the head mounted display 110 includesboth the depth sensor 280 and image sensor 290 and the control device115 includes both the depth sensor 230 and the image sensor 240.

FIG. 5 is a process flow diagram showing a method 500 for cautioning auser of a head mounted display, in accordance with an embodiment of thedisclosure.

The method 500 is performed by processing logic that includes hardware(e.g., dedicated logic, programmable logic, and microcode), software(such as software run on a general-purpose computer system or adedicated machine), or a combination of both. In one example embodiment,the processing logic resides at the head mounted display. In anotherexample embodiment, the processing logic resides at the control device.In yet another example embodiment, the processing logic resides at boththe head mounted display and the control device.

As shown in FIG. 5, the method 500 can commence at operation 510 withreceiving images and/or depth data of a scene, within which the userusing a head mounted display, is present. The scene includes apredetermined three-dimensional area. The images of the scene arecaptured by one or more sensors, for example an image sensor or a depthsensor. Thus, the scene can be defined by a field of view of the one ormore sensors. The one or more sensors can be coupled to the head mounteddisplay. The head mounted display is a wearable computer with a display,a head mounted electronic device, a head-coupled display, ahelmet-mounted display, and so forth. Alternatively, the one or moresensors are coupled to the control device.

The method 500 proceeds with tracking, at operation 520, of locationinformation of the user based at least in part on the images and/ordepth data of the scene received from the one or more sensors. Themethod 500 further proceeds with dynamically determining, at operation530, that the user approaches an edge of the scene. The determination isbased at least in part on the tracking performed at operation 520. Inone example embodiment, the edge of the scene includes one or moretangible obstacles, such as a coffee table, a chair, and so forth.

At operation 540, the user is warned that the user approaches the edgeof the scene based on the determination performed at operation 530. Thewarning can include an audio message, a visible message, turning onlight emitting diodes, gradual fading of images displayed on the headmounted display, blending displayed images in the real world view,vibrating the head mounted display, disengaging the head mounted displayor parts thereof, moving the head mounted display away from the eyes ofthe user, and the like.

In one example embodiment, the method 500 further includes receivingsecondary images of the scene from one or more secondary sensors. Theone or more secondary sensors are coupled to the control device. Incertain embodiments, the one or more secondary sensors track thelocation of the user based at least in part on the secondary images ofthe scene. Based on the secondary images, additional warnings may beprovided to the head mounted display if the user continues to move indirection of the edge of the scene. Alternately, if the user ceases tomove in the direction of the edge of the scene and moves toward thecenter of the scene, the images displayed on the head mounted displayare gradually faded into focus, or the head mounted display or partsthereof are re-engaged, the head mounted display is moved toward theeyes of the user, and the like.

FIG. 6 shows a diagrammatic representation of a computing device for amachine in the example electronic form of a computer system 600, withinwhich a set of instructions for causing the machine to perform any oneor more of the methodologies discussed herein can be executed. Inexample embodiments, the machine operates as a standalone device, or canbe connected (e.g., networked) to other machines. In a networkeddeployment, the machine can operate in the capacity of a server, aclient machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine can be a personal computer (PC), tablet PC, set-top box (STB),PDA, cellular telephone, portable music player (e.g., a portable harddrive audio device, such as a Moving Picture Experts Group Audio Layer 3(MP3) player), web appliance, network router, switch, bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that separately orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 600 includes a processor or multipleprocessors 605 (e.g., a central processing unit (CPU), a graphicsprocessing unit (GPU), or both), and a main memory 610 and a staticmemory 615, which communicate with each other via a bus 620. Thecomputer system 600 can further include a video display unit 625 (e.g.,a LCD or a cathode ray tube (CRT)). The computer system 600 alsoincludes at least one input device 630, such as an alphanumeric inputdevice (e.g., a keyboard), a cursor control device (e.g., a mouse), amicrophone, a digital camera, a video camera, and so forth. The computersystem 600 also includes a disk drive unit 635, a signal generationdevice 640 (e.g., a speaker), and a network interface device 645.

The disk drive unit 635 includes a computer-readable medium 650, whichstores one or more sets of instructions and data structures (e.g.,instructions 655) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 655 canalso reside, completely or at least partially, within the main memory610 and/or within the processors 605 during execution thereof by thecomputer system 600. The main memory 610 and the processors 605 alsoconstitute machine-readable media.

The instructions 655 can further be transmitted or received over acommunications network 660 via the network interface device 645utilizing any one of a number of well-known transfer protocols (e.g.,Hyper Text Transfer Protocol (HTTP), CAN, Serial, and Modbus). Thecommunications network 660 includes the Internet, local intranet, PAN(Personal Area Network), LAN (Local Area Network), WAN (Wide AreaNetwork), MAN (Metropolitan Area Network), virtual private network(VPN), storage area network (SAN), frame relay connection, AdvancedIntelligent Network (AIN) connection, synchronous optical network(SONET) connection, digital T1, T3, E1 or E3 line, Digital Data Service(DDS) connection, DSL (Digital Subscriber Line) connection, Ethernetconnection, ISDN (Integrated Services Digital Network) line, cablemodem, ATM (Asynchronous Transfer Mode) connection, or an FDDI (FiberDistributed Data Interface) or CDDI (Copper Distributed Data Interface)connection. Furthermore, communications also includes links to any of avariety of wireless networks including WAP (Wireless ApplicationProtocol), GPRS (General Packet Radio Service), GSM (Global System forMobile Communication), CDMA (Code Division Multiple Access) or TDMA(Time Division Multiple Access), cellular phone networks, GlobalPositioning System (GPS), CDPD (cellular digital packet data), RIM(Research in Motion, Limited) duplex paging network, Bluetooth radio, oran IEEE 802.11-based radio frequency network.

While the machine-readable medium 650 is shown in an example embodimentto be a single medium, the term “computer-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “computer-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding, or carrying a set of instructions for execution bythe machine and that causes the machine to perform any one or more ofthe methodologies of the present application, or that is capable ofstoring, encoding, or carrying data structures utilized by or associatedwith such a set of instructions. The term “computer-readable medium”shall accordingly be taken to include, but not be limited to,solid-state memories, optical and magnetic media. Such media can alsoinclude, without limitation, hard disks, floppy disks, flash memorycards, digital video disks, random access memory (RAM), read only memory(ROM), and the like.

The example embodiments described herein can be implemented in anoperating environment comprising computer-executable instructions (e.g.,software) installed on a computer, in hardware, or in a combination ofsoftware and hardware. The computer-executable instructions can bewritten in a computer programming language or can be embodied infirmware logic. If written in a programming language conforming to arecognized standard, such instructions can be executed on a variety ofhardware platforms and for interfaces to a variety of operating systems.Although not limited thereto, computer software programs forimplementing the present method can be written in any number of suitableprogramming languages such as, for example, Hypertext Markup Language(HTML), Dynamic HTML, XML, Extensible Stylesheet Language (XSL),Document Style Semantics and Specification Language (DSSSL), CascadingStyle Sheets (CSS), Synchronized Multimedia Integration Language (SMIL),Wireless Markup Language (WML), Java™, Jini™, C, C++, C#, .NET, AdobeFlash, Perl, UNIX Shell, Visual Basic or Visual Basic Script, VirtualReality Markup Language (VRML), ColdFusion™ or other compilers,assemblers, interpreters, or other computer languages or platforms.

FIG. 7 illustrates an overview of an exemplary head mounted display usedby a user for viewing and/or interacting with multimedia content. TheHMD worn by a user, in one example, allows the user to view richmultimedia content, including interactive scenes from video games,scenes from movies, internet content, and other types of interactive andnon-interactive content. The tracking of position including orientation,location, direction, etc., of the HMD is enabled by tracking a pluralityof marker elements, such as light emitting diodes, infra-red markers,visual marker elements, etc., distributed across different externalsurfaces of the HMD using a combination of sensors or based on one ormore variables used to define a volume surrounding the HMD. Some of thesensors used for tracking include, without limitation, inertial sensorswithin the HMD that allow movement tracking of the HMD, one or moreimage sensors and one or more depth sensors, wherein the image sensorsand depth sensors allow optical tracking. The tracking using inertialsensors may be enabled using one or more accelerometers and one or moregyroscopes that are disposed within the HMD. The image sensors mayinclude one or more single-lens camera, IR camera, stereo camera, etc.,and depth sensors may include one or more depth sensing cameras,ultrasonic camera, three-dimensional (3D) stereo cameras, video cameras,etc. The image sensors and depth sensors encompass one or more camerasprovided within the HMD as well as external cameras that are dispersedwithin a real-world scene of the user wearing the HMD. The imagessensors and/or the depth sensors within the HMD, for example, are usedto capture images/videos of the real-world objects/scenes in theimmediate vicinity of the user from the perspective of the user wearingthe HMD. The captured images/videos may be rendered in the displayportion of the HMD when content is being adjusted, as will be explainedfurther below.

The image and/or depth sensors distributed externally within the sceneof the user, for example, are configured to capture images/videos of thevarious markers, such as lights, light emitting diodes (LEDs), infra-redmarkers, etc., distributed across the external surface of the HMD 110.In one embodiment, the images/videos are transmitted to a client systemwhere the images/video frames are analyzed to accurately determine thelocation of the HMD. Additionally, the images/videos are analyzed withinthe HMD to determine the location, orientation, direction of the HMDwith reference to other objects in the real-world scene. In oneconfiguration, the HMD includes LEDs (for e.g., represented by bubbles2, 4 and 6) disposed at strategic locations on one or more externalsurfaces of the HMD. For instance, the LEDs may be disposed on the fourcorners of the front block unit 110 a (e.g., also referred to herein asthe optics block) of the HMD and two on the rear/back section 110 b ofthe HMD 110. The rear section 110 b is disposed on an adjustable bandunit. The HMD may also include other surfaces, such as 110 c and 110 dto allow the user to safely and securely harness the HMD on the user'shead. In some embodiments, the front LEDs are configured to be partiallydisposed on the front surface and partially on side surfaces that aredisposed on each side of the front surface, to define a partial L-shape,a curved L-shape, a boomerang shape, a curved rectangle, a curved line,a spot, a circle, a pattern, or combinations thereof. The markers arenot restricted to LEDs but can also include lights, infra-red markers,color coded markers, reflective markers, etc. The analysis of theimages/video frames are used to compute relative distance of thedifferent markers of the HMD from one another and from one or morereference points. The computed distances are used to determine a volumearound the HMD and changes to the volume during use to more accuratelydefine the HMD's position. Additionally, the video frames/imagescaptured by the various sensors are analyzed to determine positionincluding orientation, location, direction of movement of the variousmarkers to more accurately determine the direction of movement orposition of the user wearing the HMD. It should be noted that the volumesurrounding the HMD may remain the same while the location, orientationand direction of the various markers may change during use, for example,when the user wearing the HMD is spinning around. The analysis of thevideo frames/images will provide a more accurate measure of thelocation, direction and orientation of the HMD. As a safety precaution,the movement of the HMD is analyzed. When the movement exceeds a certainpre-defined threshold value or is determined to be unsafe (e.g., user isfalling backwards), a signal can be generated to cause interruption ofthe interactive scenes currently being rendered on a display screen ofthe HMD. Alternately, some other form of warning may be generated tosignal to the user that they may be moving in an unsafe manner.

The HMD is configured to provide a view into an interactive scene of avideo game, for example, at the display portion of the HMD. The userwearing the HMD will be able to move his or her head in any direction toview other parts of the interactive scene and provide inputs to moveabout in the rendered virtual interactive scene, based on the user'shead movement. In one embodiment, the virtual interactive scene isrendered in a rich three dimensional (3D) format. In one configuration,the HMD is configured to communicate with a client computer system 106,which renders the content presented to the HMD. The content (e.g., game,movie, video, audio, images, multimedia, etc.), in anotherconfiguration, may be streamed from a remote server or servers to theHMD using cloud gaming infrastructure 112. In some examples, the contentfrom the remote server(s) is downloaded to a client system and thentransferred to the HMD for rendering. In other examples, the content isdirectly streamed from the remote server(s) to the HMD for rendering.

When the user wearing the HMD is facing an image/depth sensor camera,the camera should be able to view all LEDs and/or markers disposed inthe front face of the HMD 110. Alternately, the camera should be able totrack at least one LED and/or maker disposed in the front face of theHMD 110. A geometric model of the HMD available at the client system,for example, can be accessed by programs, such as game or otherprograms, executing on the client system, to determine depth (relativeto the camera) and orientation of the user's head when wearing the HMD.For instance, four LEDs disposed on the corners (e.g., outlining arectangular shape) of the front surface of the HMD and two LEDs disposedin the back surface of the HMD may be tracked by capturing images/videoframes to determine if the user is viewing down, up or to the sides.Other marker elements, in addition to or instead of the LEDs, disposedacross different external surfaces of the HMD may also be used todetermine the orientation, direction, location, etc., of the HMD worn bythe user based on the user's actions in the form of head movement.

FIG. 8 illustrates an example of a user wearing the HMD 110, during use,in accordance with one embodiment. In this example, the position of theHMD is tracked using image data 802 extracted from images/video framescaptured by the camera 108. In some embodiments, in addition to trackingthe position of the HMD, a controller held by the user and used toprovide input to the content rendered on the HMD, may also be tracked.As illustrated in FIG. 8, the controller is tracked using image data 804extracted from captured video frames by the camera 108. FIG. 8 shows anexemplary configuration where the HMD is connected to the computingsystem (i.e., client device) 106 via a cable 806. Cable 806 is capableof transmitting rich media data and signals between the HMD and theclient computing system 106. In one embodiment, the HMD obtains powerthrough the cable 806. In an alternate embodiment, the power for the HMDcan be obtained via another cable connected to a power source. In stillanother embodiment, the HMD can have a battery that is rechargeable, soas to avoid extra power cords.

FIG. 9 illustrates an embodiment of a system that is used for providingwarning to a user (or player) when the user comes close to a real-worldobject, an edge of a scene, enters other player's game zone, orturns/moves away from the client device, based on user actions whilewatching interactive scenes on the display portion of the HMD. Thesystem may also be used to adjust content rendered on the displayportion of the HMD based on the user's actions. The embodiment shown inFIG. 9 illustrates two players engaged in interactive gameplay of avideo game. The interactive scenes from the video game are rendered onthe display portion of the respective HMDs of the two players. Theinteractive scenes for the video game are provided by a client device,such as a game console (not shown in FIG. 9), where the game isexecuting or is downloaded from a game server 204 (where the game isexecuting) via the network 202 to the game console and rendered on theHMDs. Alternately, the interactive scenes may be streamed directly fromthe game server 204 via the network 202 to the HMDs 110′. Theinteractive scenes are rendered in the display portion of the respectiveHMDs.

The position of the HMDs and the controllers handled by the players aretracked by one or more image sensors, inertial sensors and/or depthsensors. The images/videos capture the respective player's actions, suchas movement of the hand(s) operating a controller, change in positionincluding location, direction and orientation, of the HMD and/or thecontroller. The captured images/video frames are analyzed to detectposition of the HMD and/or the controller of each player by determininglocation of the various markers, such as markers defined by LEDs (1, 2,3, 4 (not shown), 5 and 6) on each of the HMDs. In one embodiment, aprocessor within the client device, such as a game console 106, receivesthe captured images/video frames of the markers disposed on each of theHMDs and analyzes the content captured therein to identify the positionof the HMD based on the captured images/video frames of the markersdisposed on the respective HMDs. Similar analysis of the markersdisposed on the controller may also be carried out to correctlydetermine the position of the controller. In some embodiments, aprocessor in each of the HMD may be used to provide additional data withrespect to the movement of the respective HMDs independently and inrelation to one another. Accordingly, the processor of the HMD gathersthe images/video data and transmits the same to the client device 106where the analysis is performed. In some embodiments, the relativeposition of a first HMD with respect to a second HMD may also besimilarly computed by obtaining and analyzing the image/video frames ofthe markers on the other HMD and computing the relative position of theHMD worn by the first player with respect to the HMD worn by the secondplayer. Based on the images/video frames, game zone of each player A andB may be established, as illustrated in FIG. 9.

The client device 106 (also referred to, in some embodiments, byreference numeral 120 (FIG. 1)) includes a plurality of modules that areused in analyzing the data within the image/video frames captured by thevarious sensors, identifying appropriate content to be rendered at thedisplay portion of the HMD and transmitting the appropriate content tothe HMD. For instance, a HMD data processing module 905 in the clientdevice 106 is used in analyzing the images/video frames received fromthe various sensors and determining the position of the HMD and/or thecontroller worn/held by a user. Analysis information from the HMD dataprocessing module 905 is shared with a warning processing module 906within the client device 106. In one embodiment, the information fromthe analysis may identify the proximity of a player to a real-worldobject, proximity of the player to an edge of a scene, proximity of aplayer to another player, etc. For instance, in the example illustratedin FIG. 9, during interaction with the video game, player A may intrudeinto player B's game zone or vice versa. In this instance, the variousimages/videos of the two players captured by the sensors distributedwithin the respective HMDs and throughout the real-world scene of thetwo players are analyzed to determine the extent of intrusion of playerA into player B's game zone or vice versa. Depending on the actions ofthe players in the video game, such intrusion may result in first playerpotentially injuring the second player or vice versa. In order toprevent any harm to the players and to provide a safe playingexperience, the analysis information of the images/video frames may beused by the warning processing 906 to generate a signal for warning therespective players.

In addition to the HMD processing module analyzing the images/videosinformation, a game logic module 901 may also analyze the images/videosand share the analysis information with a game action trigger module902. In this instance, the HMD processing module 905 shares theimages/video information with the game logic module 901. The analysis bythe game logic module may be specific to the video game currentlyexecuting and may be performed with reference to the game context. Forexample, the game logic module may analyze the images/video framesreceived at the client device with respect to the game intensitycontext. Information from the game logic module 901 may be used by thegame action trigger module 902 to initiate an appropriate game specifictrigger action. The game specific trigger action may include a flagidentifying that it is a high-intensity game or a low-intensity game.The game specific trigger action may be provided to the warningprocessing module 906 for generating an appropriate warning signal fortransmitting to the HMD.

In one embodiment, the images/video frames captured by the varioussensors are transmitted to the game server 204 over the network 202where the images/video frames are analyzed and the analyzed data istransmitted to the game logic module 901 for sharing with the gameaction trigger 902. Based on the analyzed data, the game action trigger902 determines which action trigger to initiate.

In addition to the game specific analysis performed by the game logicmodule 901, a system data processing module 903 may also be used toanalyze the images/video frames received from the HMD data processingmodule to provide system specific action trigger. For example, thesystem data processing module 903 may analyze the images/video frames inthe context of the system setting and provide appropriate data to thesystem action trigger module 904. The system action trigger 904determines which action trigger to initiate. Data from the system actiontrigger 904 is also transmitted to the warning processing module 906.The warning processing module 906 receives the game specific actiontriggers and system specific action triggers as well as the data fromthe HMD data processing module 905 and generates a warning signal forthe HMD processor. The warning signal generated by the warningprocessing module 906 is transmitted through the HMD data processingmodule 905 to the HMD. In the HMD data processing module 905, the signalis used to determine if any adjustment to the content needs to be madeat the display portion of the HMD of either player A or player B orboth. If any adjustment to the content needs to be made, the HMD dataprocessing module 905 will transmit appropriate data content tointerrupt the interactive scenes that are being rendered on the displayportion of the HMD. In some embodiments, the appropriate data contentmay convey a warning to the player wearing the HMD. The warning may beto inform the player of the real-world environmental condition detectednear the player based on the movement of the HMD worn by the user. Insome embodiments, the warning may inform the player of the severity ofthe real-world environmental safety condition detected near the playerbased on the movement of the HMD worn by the player. For example, ifplayer A moves into the game zone of player B, then the warning signalmay be directed to the player A's HMD to let the player A know thathe/she is intruding into player B's game zone. The warning signal may bein the form of a textual warning that is rendered at the display portionof the player A'HMD, as illustrated in FIG. 9, while player B's HMD willcontinue to render the interactive scenes of the video game. In oneembodiment, depending on the intensity of the video game, the textualwarning may be rendered at the player A's HMD in the foreground whilethe interactive scenes of the video game continue to render in thebackground. In this embodiment, the brightness level of the interactivescenes may be faded and the textual warning presented brightly. In oneexample, the fading is done in a graduated manner depending on theintensity of the game so as to avoid disorienting the player. Inaddition to or in place of the textual warning, the warning may beprovided in haptic format, audio format, etc.

FIG. 10 illustrates an exemplary embodiment for determining position ofthe HMD and/or the controller. In this embodiment, the one or more imagesensors and/or depth sensors are used to determine a volume around theHMD worn by a user and or around a controller handled by the user. Thevolume around the HMD, for example, may be determined using a depthsensor 108, such as a stereo camera, that tracks the location of thevarious marker elements disposed on the external surface of the HMD wornby the player, in images/videos and associates a depth aspect to theimages/videos. Volume around the controller may also be similarlydetermined. As shown in FIG. 10, the depth sensor 108 may determine avolume 303 around the HMD and a volume 305 around the controller. Thevolume may be used to more precisely determine the position of the HMDand/or the controller handled by a player.

FIGS. 10A and 10B illustrate how an HMD data processing module uses thevolume around the HMD and/or the controller to determine a position,such as orientation, location, direction of the user wearing/handlingthe HMD/the controller. The HMD data processing module may determine ifa player (i.e., user) is sitting or standing when playing the videogame. A video game may have certain restrictions on the position of theplayer. For example for a high intensity game, the video game mayrequire the player to be seated during gameplay in order to avoid playerdisorientation. When a player starts out playing the video game whilesitting and during the course of the gameplay stands, the gameprocessing module may be able to detect a change in the position of theuser by determining a change in elevation of the HMD and/or thecontroller. Based on the game's restrictions and the change in theelevation of the HMD and/or the controller, a warning signal may begenerated. To determine the change in the elevation, the initialposition of the HMD worn by the player is computed. As shown in FIG.10A, the image sensors/depth sensors determine one or more variablesdefining a volume 303 surrounding the HMD. For example, the imagesensors/depth sensors may be used to determine a height h_(h) (whereinh_(h)=h4−h3), depth (d_(h)) and width w_(h) (wherein w_(h)=w1−w2) for aregion surrounding the HMD and the volume around the HMD may be computedas a function of h_(h), d_(h) and w_(h). Similarly, the volumesurrounding a controller may be determined by determining a height h_(c)(wherein h_(c)=h2−h1), depth d_(c) and width w_(c) (wherein w_(c)=w1−w2)of a region surrounding the controller 104 and the volume 305 around thecontroller 104 may be computed as a function of h_(c), d_(c) and w_(c).For simplicity purpose, the width of the region surrounding thecontroller is shown to be same as the width of the region surroundingthe HMD. It should be noted that the variables (i.e., width, height anddepth) for the volume surrounding the controller may be different thanthe variables for the volume surrounding the HMD.

When a player changes his position by sitting, for example, asillustrated in FIG. 10B, the change in elevation of the HMD is detectedby the image/depth sensors and a new volume is computed for the HMD bythe HMD data processing module. In this example, the variable related tothe height has changed and the new height h_(h)′ (whereinh_(h)′=h4′−h3′) of the HMD is computed using the data captured by theimage/depth sensors and the new volume of the HMD is computed as afunction of w_(h), d_(h), h_(h)′. In this example, the volume around thecontroller (defined by height h1, h2 used for computing h_(c), and widthw1, w2 used for computing w_(c)) may have stayed the same. In alternateexamples, the volume around the controller may also have changed and asa result a new volume for the controller may be computed using thechange in the respective height and/or width of the controller. Based onthe change in the position of the player captured by the images/videoframes, the HMD data processing module 905 may send data to adjust thecontent rendered on the HMD and/or provide warning for the player. Thewarning may include a request for the player to change hislocation/position, etc.

FIG. 10C illustrates computing change in position of the HMD/controllerbased on the player's action, in one embodiment of the invention. Anexternal camera 115, such as depth sensing camera, video camera, etc.,may be used to track the relative position of the HMD worn by a playerto the controller held by the player at various times. The HMD dataprocessing module uses the images from the camera 115 to detect a changein position of the HMD/controller. A change is detected by computing theinitial position of the HMD/controller at time t0 (defined by ‘h₁’) andthe position of the HMD/controller at time t1 (defined by ‘h₂’). The HMDdata processing module further determines if the difference between thetwo position (h₁−h₂) exceeds a pre-defined threshold value. Thepre-defined threshold value may be set to allow some variations in theposition of the HMD worn by the player that may be caused by the playerlooking up, down or sideways.

FIGS. 11A-11E illustrate an exemplary embodiment wherein a user's changein orientation may result in a change in environmental safety conditionin the vicinity of the user wearing the HMD. The change in HMD'sorientation may be tracked and warning signals generated and/or datacontent transmitted to interrupt the interactive scenes rendering on theHMD of the player, based on the severity of the environmental safetycondition detected in the vicinity of the user determined by thetracking. For example FIG. 11A shows the user wearing the HMDinteracting with the video game executing on a game console 106, usingthe controller. The interactive scenes of the video game are rendered ona display portion of the HMD. User interactions at the controller aretransmitted wirelessly to the game console 106 and the HMD movement isprocessed by a processor in the HMD and the processed contenttransmitted through the cable 806 to the game console 106. The inputfrom the controller and the HMD are used as user input to the videogame. The game console updates the interactive game scenes rendered atthe HMD based on the input provided by the user. It should be noted thatin the various embodiments, a user and a player are used interchangeablyto refer to a person who is actively interacting with content presentedon a display portion of the HMD. The content may be interactive gamescenes of a video game, an interactive application, a social interactivestream, etc. The HMD worn by the user, in FIG. 11A, may be tracked usingmarker elements, such as LEDs (represented by reference numerals 1-6,for example), visual markers, infra-red markers, etc., and it may bedetermined from the tracking that the user wearing the HMD is facingforward toward the image/depth sensors 115 of the system (for e.g., gameconsole 12) executing the video game, at time t0. During the gameplay,the user may start to spin around. As a result of the spinning, the usermay turn right at time t1, as illustrated in FIG. 11B. The cable 806connecting the HMD to the game console/client device slowly begins towrap around the user, leading to a potential safety hazard to the user.As the user continues to spin, the user turns right and faces backwardat time t2 away from the image/depth sensors of the system, asillustrated in FIG. 11C, and face left at time t3, as illustrated inFIG. 11D. The cable 806 continues to wrap around the user. If the usercontinues to spin around, the cable 806 may continue to wrap around theuser's neck and/or body, as illustrated in FIG. 11E, wherein it is shownthat the cable has wrapped twice around the neck of the user, that maycause potential harm to the user. The user may not even be aware of thiswrapping as he/she may be totally immersed in the video game. Further,as can be seen in FIG. 11E, the wrapping of the cable causes the cableto become taut. In order to prevent the cable from becoming a safetyhazard to the user, causing bodily harm to the user, and/or resulting inthe un-plugging of the HMD, the HMD data processing module may detectthe change in position of the HMD, identify the environmental safetycondition, and automatically provide appropriate warnings and/or data toadjust/interrupt content rendered on the HMD's display portion. Thespinning action, for example, may be detected by the inertial sensors(one or more gyroscopes and one or more accelerometers) of the HMD. TheHMD data processing module may analyze the positional data of the HMDworn by the user to adjust content and/or provide warning messages.

FIG. 11F illustrates yet another hazard that may befall a user when theuser performs certain actions while still immersed in gameplay of thevideo game, in one exemplary embodiment. In this embodiment, a userplaying the video game (at time t0) may turn around and being to walkaway from the game console, (as shown by user's position at time t1 andt2 in FIG. 11F) while continuing to interact with the video game. Theposition of the player is tracked using the LEDs, for example, disposedaround the external surfaces of the HMD. In the embodiment illustratedin FIG. 11F, the user's initial position is determined by tracking theLEDs 1-6 disposed on the front portion 110 a and back portion 110 b ofthe HMD. Although FIG. 11F shows only LEDs 2 and 4 on the front portion110 a, it should be noted that the LEDs 1 and 3 disposed on the otherside of the HMD are also tracked. The user's movement away from the gameconsole may be determined by the change in the position of the LEDs 1-6on the HMD surface. For example, as the LEDs 2, 4, 6 move away from thecamera's 108 view other markers, such as LEDs 1, 3 and 5, come into theview of the camera 108. By tracking the LEDs disposed on the externalsurfaces of the HMD, it may be determined that the user has turned awayfrom facing the camera to facing away from the camera and walking awaytoward a set of stairs, for example, while completely immersed ingameplay. The HMD data processing module may detect change in datarelated to the HMD, such as change in position, elevation, orientation,direction, location of the various LEDs on the HMD worn by the user. Thechanged data may be used to determine the position of the user andprovide appropriate warning messages. For example, at time t0, no changein the position of the user wearing the HMD is detected and, as aresult, interactive game scene from the video game continues to berendered at the display portion of the HMD. At time t1 the HMD dataprocessing module detects the user turning and walking away from theconsole and, in response to the user action, generates a warning signalthat identifies a severity level of the environmental safety and sendsappropriate data for rendering on the HMD. When the warning signalidentifies a low severity level, a warning message that is descriptiveof the severity level, is presented to augment the interactive gamescene rendered on the HMD. At time t2, as the user continues to walkaway toward the stairs, the HMD data processing module determines theseverity of the environmental safety condition in the user's vicinity,and transmits images of the real-world objects from the user's immediatevicinity as well as appropriate warning message indicating high severitylevel, for rendering on the HMD. Due to the high severity level, thetransmission of the images causes interruption to the interactive gamescenes rendered on the HMD and rendition of the images of the real-worldobjects on the HMD. In addition, a warning message defining the severityof the environmental safety condition is also rendered.

The change in condition, (i.e., position, elevation, orientation,direction, location) of the HMD worn by the user is not restricted totracking LEDs but may include tracking the movement and/or rotation ofmarker elements, such as lights, visual markers, infrared markerelements, etc., disposed on the different external surfaces of the HMDand the controller, using one or more image sensors, depth sensorsand/or inertial sensors (i.e., one or more gyroscopes, one or moreaccelerometers) or any combinations thereof. When a change in theposition of the HMD and/or the controller is detected that exceeds apre-defined threshold value, the HMD data processing module may bepre-programmed to stop the video game, pause the game, interrupt thegame play/interactive scenes rendered on the HMD, and render the datarelated to the environment in the vicinity of the user wearing the HMD.

It should be noted that the change in condition may be monitored by boththe operating system code of the HMD that initiates a system specificaction trigger and the game code that initiates a game specific actiontrigger. Accordingly, the warning message generated is in accordance tothe system specific and game specific action triggers. Thus, the variouscodes/code modules involved in the execution of the video game, such asgame code module, code from the HMD's operating system (OS), applicationcode from the HMD, a combination of code from HMD/OS, firmware, softwareon a game console, code executed by the game console or applicationserver, or code executing on a cloud server, etc., monitor the change incondition (i.e., orientation, position, location, direction, etc.) ofthe HMD and/or the controller worn/handled by the user to generateappropriate warning message and to transmit appropriate data that isused to warn the user and to adjust the content rendered on the HMD. Thetransmitted data can be used to adjust the content rendered on the HMDgradually to enable a user to fully immerse or slowly release from thevideo game to avoid disorienting the user. The warning message may be inthe form of an audio signal, a sound signal, a textual signal, agraphical signal (animated or static), or any other form of signal thatcan be provided to the user warning the user of safety issues during useof the HMD/controller. If the warning signals are not sufficient toprevent the safety concern from occurring, the HMD may be automaticallytransitioned out of the interactive scene currently rendering on theHMD.

With the detailed description of the various embodiments, a method foradjusting content and/or providing warning to a user will now bedescribed with reference to FIG. 12. The method begins at operation1210, wherein a game is executed. The game may be executed on a clientdevice, on a game console, or on a game server in a game cloud. Theexecution of the game results in the interactive scenes of the game tobe streamed to a head mounted display (HMD) worn by a user, forrendering. A change in the position of the HMD worn by the user isdetected while the user is interacting with the game, as illustrated inoperation 1220. The change in position may be detected by monitoring thelocation, orientation, direction of one or more markers distributedthroughout various external surfaces of the HMD using any one orcombination of image sensors, depth sensors or inertial sensors.Alternately, the change in position may be detected using data obtainedfrom inertial sensors distributed within the HMD. When a change in aposition of the HMD is detected, the change is evaluated, as illustratedin operation 1230. The evaluation of the change may be performed by codemodules within the HMD, by game code module of a game executing on aclient device that provides data for rendering the interactive scenes ofthe game at the HMD, by an operating system of the HMD, of the clientdevice, or of a game cloud server. As described earlier, the evaluationof the change may include determining if the change in the positionexceeds a pre-defined threshold value. If the change in position of theHMD exceeds the pre-defined threshold value, a signal is generated bythe code module at the client device. Based on the signal generated atthe client device, data is transmitted to the HMD to cause interruptionand/or adjustment of the content rendered on the HMD, as illustrated inoperation 1240. The adjustment to content may be to provide a warningmessage to a user wearing the HMD alongside the interactive game scenecontent, adjust content to fade in brightness, transition out of thecontent, render images of real-world objects/scenes from the vicinity ofthe user instead of the interactive game scenes, etc. The adjustment tothe content can be graduated to allow the user to extricate from thevideo game or slowly immerse back into the video game, after having beeninterrupted, depending on the signal.

Thus, the technology for warning users of head mounted displays isdisclosed. The various embodiments are able to detect elevationpositions of the HMD and/or the controller in a plurality of ways,including height comparisons of the HMD (e.g., LEDs), controller LEDs,motions by the controller and/or LEDs. In some cases, some games, suchas high intensity video games, may require that the user sit whileplaying the game. For such games, the system can monitor to determinewhen the user stands up while playing the game. Such monitoring can beenabled by monitoring the change in a variable, such as height, andissue appropriate warning or interruption to the interactive gamescenes. The system can also determine when a cord of the HMD is beingpulled too many times, too tight, or is wrapped around a user, for e.g.,by detecting motion of the HMD using gyroscope, accelerometer, rotationof the LEDs of the HMD, or LEDs of the controller, or combinations oftwo or more thereof. Once such signal or condition is detected thesystem provides appropriate warnings and/or data to transition out ofthe game scenes. The condition is monitored by operating system code ofthe HMD, code of a particular game, software on a game console, codeexecuted by the game console, code executing of the game, or codeexecuting on a cloud server.

Although embodiments have been described with reference to specificexample embodiments, it will be evident that various modifications andchanges can be made to these example embodiments without departing fromthe broader spirit and scope of the present application. Accordingly,the specification and drawings are to be regarded in an illustrativerather than a restrictive sense.

FIG. 13 illustrates hardware and user interfaces that may be used toimplement some embodiments of the invention. FIG. 6 schematicallyillustrates the overall system architecture of the Sony® PlayStation 3®entertainment device. Other versions of PlayStation may include more orless features. A system unit 1300 is provided, with various peripheraldevices connectable to the system unit 1300. The system unit 1300includes: a Cell processor 1302; a Rambus® dynamic random access memory(XDRAM) unit 1304; a Reality Synthesizer graphics unit 1306 with adedicated video random access memory (VRAM) unit 1308; and an I/O bridge1310. The system unit 1300 also comprises a Blu Ray® Disk BD-ROM®optical disk reader 1312 for reading from a disk 1312 a and a removableslot-in hard disk drive (HDD) 1314, accessible through the I/O bridge1310. Optionally, the system unit 1300 also comprises a memory cardreader 1301 for reading compact flash memory cards, Memory Stick® memorycards and the like, which is similarly accessible through the I/O bridge1310.

The I/O bridge 1310 also connects to six Universal Serial Bus (USB) 2.0ports 1316; a gigabit Ethernet port 1318; an IEEE 802.11b/g wirelessnetwork (Wi-Fi) port 1320; and a Bluetooth® wireless link port 1322capable of supporting of up to seven Bluetooth connections.

In operation, the I/O bridge 1310 handles all wireless, USB and Ethernetdata, including data from one or more game controllers 110 and 1324. Forexample, when a user is playing a game, the I/O bridge 1310 receivesdata from the game controller 110 and 1324 via a Bluetooth link anddirects it to the Cell processor 1302, which updates the current stateof the game accordingly.

The wireless, USB and Ethernet ports also provide connectivity for otherperipheral devices in addition to game controllers 110 and 1324, suchas: a remote control 1326; a keyboard 1328; a mouse 1330; a portableentertainment device 1332 such as a Sony PSP® entertainment device; avideo camera such as a PlayStation® Eye Camera 1334; a shape object1336; and a microphone 1338. Such peripheral devices may therefore inprinciple be connected to the system unit 1300 wirelessly; for example,the portable entertainment device 1332 may communicate via a Wi-Fiad-hoc connection, while the shape object 1336 may communicate via aBluetooth link.

The provision of these interfaces means that the PlayStation 3 device isalso potentially compatible with other peripheral devices such asdigital video recorders (DVRs), set-top boxes, digital cameras, portablemedia players, Voice over Internet Protocol (IP) telephones, mobiletelephones, printers and scanners. In addition, a legacy memory cardreader 1340 may be connected to the system unit via a USB port 1316,enabling the reading of memory cards of the kind used by thePlayStation® or PlayStation 2® devices.

The game controllers 110 and 1324 are operable to communicate wirelesslywith the system unit 1300 via the Bluetooth link, or to be connected toa USB port, thereby also providing power by which to charge the batteryof the game controllers 110 and 1324. Game controllers 110 and 1324 canalso include memory, a processor, a memory card reader, permanent memorysuch as flash memory, light emitters such as an illuminated sphericalsection, light emitting diodes (LEDs), or infrared lights, microphoneand speaker for ultrasound communications, an acoustic chamber, adigital camera, an internal clock, a recognizable shape facing the gameconsole, and wireless communications using protocols such as Bluetooth®,WiFi™, etc. The recognizable shape can be in a shape substantially of asphere, a cube, parallelogram, a rectangular parallelepiped, a cone, apyramid, a soccer ball, a football or rugby ball, an imperfect sphere, asection of a sphere, a truncated pyramid, a truncated cone, a baseballbat, a truncated cube, a polyhedron, a star, etc., or a combination oftwo of more of these shapes.

Game controller 1324 is a controller designed to be used with two hands,and game controller 110 is a single-hand controller with a ballattachment. In addition to one or more analog joysticks and conventionalcontrol buttons, the game controller is susceptible to three-dimensionallocation determination. Consequently gestures and movements by the userof the game controller may be translated as inputs to a game in additionto or instead of conventional button or joystick commands Optionally,other wirelessly enabled peripheral devices such as the Sony PSP®portable device may be used as a controller. In the case of the SonyPSP® portable device, additional game or control information (forexample, control instructions or number of lives) may be provided on thescreen of the device. Other alternative or supplementary control devicesmay also be used, such as a dance mat (not shown), a light gun (notshown), a steering wheel and pedals (not shown) or bespoke controllers,such as a single or several large buttons for a rapid-response quiz game(also not shown).

The remote control 1326 is also operable to communicate wirelessly withthe system unit 1300 via a Bluetooth link. The remote control 1326comprises controls suitable for the operation of the Blu Ray™ DiskBD-ROM reader 1312 and for the navigation of disk content.

The Blu Ray™ Disk BD-ROM reader 1312 is operable to read CD-ROMscompatible with the PlayStation and PlayStation 2 devices, in additionto conventional pre-recorded and recordable CDs, and so-called SuperAudio CDs. The reader 1312 is also operable to read DVD-ROMs compatiblewith the Playstation 2 and PlayStation 3 devices, in addition toconventional pre-recorded and recordable DVDs. The reader 1312 isfurther operable to read BD-ROMs compatible with the PlayStation 3device, as well as conventional pre-recorded and recordable Blu-RayDisks.

The system unit 1300 is operable to supply audio and video, eithergenerated or decoded by the PlayStation 3 device via the RealitySynthesizer graphics unit (RSX) 1306, through audio and video connectorsto a display and sound output device 1342 such as a monitor ortelevision set having a display 1346 and one or more loudspeakers 1348,or stand-alone speakers 1350. In one embodiment, voice and gaze inputsare utilized to play sound toward specific audio speakers according tothe POG of the user. The audio connectors 1358 may include conventionalanalogue and digital outputs while the video connectors 1360 mayvariously include component video, S-video, composite video and one ormore High Definition Multimedia Interface (HDMI) outputs. Consequently,video output may be in formats such as PAL or NTSC, or in 720p, 1080i or1080p high definition.

Audio processing (generation, decoding and so on) is performed by theCell processor 1302. The PlayStation 3 device's operating systemsupports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), andthe decoding of 7.1 surround sound from Blu-Ray® disks.

In the present embodiment, the video camera 1334 comprises a singleCharge Coupled Device (CCD), an LED indicator, and hardware-basedreal-time data compression and encoding apparatus so that compressedvideo data may be transmitted in an appropriate format such as anintra-image based MPEG (motion picture expert group) standard fordecoding by the system unit 1300. The camera LED indicator is arrangedto illuminate in response to appropriate control data from the systemunit 1300, for example to signify adverse lighting conditions.Embodiments of the video camera 1334 may variously connect to the systemunit 1300 via a USB, Bluetooth or Wi-Fi communication port. Embodimentsof the video camera may include one or more associated microphones andalso be capable of transmitting audio data. In embodiments of the videocamera, the CCD may have a resolution suitable for high-definition videocapture. In use, images captured by the video camera may, for example,be incorporated within a game or interpreted as game control inputs. Inanother embodiment, the camera is an infrared camera suitable fordetecting infrared light.

In general, in order for successful data communication to occur with aperipheral device such as a video camera or remote control via one ofthe communication ports of the system unit 1300, an appropriate piece ofsoftware such as a device driver should be provided. Device drivertechnology is well-known and will not be described in detail here,except to say that the skilled man will be aware that a device driver orsimilar software interface may be required in the present embodimentdescribed.

FIG. 14 is a block diagram of a Game System 1100, according to variousembodiments of the invention. Game System 1100 is configured to providea video stream to one or more Clients 1110 via a Network 1115. GameSystem 1100 typically includes a Video Server System 1120 and anoptional game server 1125. Video Server System 1120 is configured toprovide the video stream to the one or more Clients 1110 with a minimalquality of service. For example, Video Server System 1120 may receive agame command that changes the state of or a point of view within a videogame, and provide Clients 1110 with an updated video stream reflectingthis change in state with minimal lag time. The Video Server System 1120may be configured to provide the video stream in a wide variety ofalternative video formats, including formats yet to be defined. Further,the video stream may include video frames configured for presentation toa user at a wide variety of frame rates. Typical frame rates are 30frames per second, 60 frames per second, and 1120 frames per second.Although higher or lower frame rates are included in alternativeembodiments of the invention.

Clients 1110, referred to herein individually as 1110A., 1110B., etc.,may include head mounted displays, terminals, personal computers, gameconsoles, tablet computers, telephones, set top boxes, kiosks, wirelessdevices, digital pads, stand-alone devices, handheld game playingdevices, and/or the like. Typically, Clients 1110 are configured toreceive encoded video streams, decode the video streams, and present theresulting video to a user, e.g., a player of a game. The processes ofreceiving encoded video streams and/or decoding the video streamstypically includes storing individual video frames in a receive bufferof the client. The video streams may be presented to the user on adisplay integral to Client 1110 or on a separate device such as amonitor or television. Clients 1110 are optionally configured to supportmore than one game player. For example, a game console may be configuredto support two, three, four or more simultaneous players. Each of theseplayers may receive a separate video stream, or a single video streammay include regions of a frame generated specifically for each player,e.g., generated based on each player's point of view. Clients 1110 areoptionally geographically dispersed. The number of clients included inGame System 1100 may vary widely from one or two to thousands, tens ofthousands, or more. As used herein, the term “game player” is used torefer to a person that plays a game and the term “game playing device”is used to refer to a device used to play a game. In some embodiments,the game playing device may refer to a plurality of computing devicesthat cooperate to deliver a game experience to the user. For example, agame console and an HMD may cooperate with the video server system 1120to deliver a game viewed through the HMD. In one embodiment, the gameconsole receives the video stream from the video server system 1120, andthe game console forwards the video stream, or updates to the videostream, to the HMD for rendering.

Clients 1110 are configured to receive video streams via Network 1115.Network 1115 may be any type of communication network including, atelephone network, the Internet, wireless networks, powerline networks,local area networks, wide area networks, private networks, and/or thelike. In typical embodiments, the video streams are communicated viastandard protocols, such as TCP/IP or UDP/IP. Alternatively, the videostreams are communicated via proprietary standards.

A typical example of Clients 1110 is a personal computer comprising aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logic mayinclude hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams arewell known in the art and vary depending on the particular encodingscheme used.

Clients 1110 may, but are not required to, further include systemsconfigured for modifying received video. For example, a client may beconfigured to perform further rendering, to overlay one video image onanother video image, to crop a video image, and/or the like. Forexample, Clients 1110 may be configured to receive various types ofvideo frames, such as I-frames, P-frames and B-frames, and to processthese frames into images for display to a user. In some embodiments, amember of Clients 1110 is configured to perform further rendering,shading, conversion to 3-D, or like operations on the video stream. Amember of Clients 1110 is optionally configured to receive more than oneaudio or video stream. Input devices of Clients 1110 may include, forexample, a one-hand game controller, a two-hand game controller, agesture recognition system, a gaze recognition system, a voicerecognition system, a keyboard, a joystick, a pointing device, a forcefeedback device, a motion and/or location sensing device, a mouse, atouch screen, a neural interface, a camera, input devices yet to bedeveloped, and/or the like.

The video stream (and optionally audio stream) received by Clients 1110is generated and provided by Video Server System 1120. As is describedfurther elsewhere herein, this video stream includes video frames (andthe audio stream includes audio frames). The video frames are configured(e.g., they include pixel information in an appropriate data structure)to contribute meaningfully to the images displayed to the user. As usedherein, the term “video frames” is used to refer to frames includingpredominantly information that is configured to contribute to, e.g. toeffect, the images shown to the user. Most of the teachings herein withregard to “video frames” can also be applied to “audio frames.”

Clients 1110 are typically configured to receive inputs from a user.These inputs may include game commands configured to change the state ofthe video game or otherwise affect game play. The game commands can bereceived using input devices and/or may be automatically generated bycomputing instructions executing on Clients 1110. The received gamecommands are communicated from Clients 1110 via Network 1115 to VideoServer System 1120 and/or Game Server 1125. For example, in someembodiments, the game commands are communicated to Game Server 1125 viaVideo Server System 1120. In some embodiments, separate copies of thegame commands are communicated from Clients 1110 to Game Server 1125 andVideo Server System 1120. The communication of game commands isoptionally dependent on the identity of the command Game commands areoptionally communicated from Client 1110A through a different route orcommunication channel that that used to provide audio or video streamsto Client 1110A.

Game Server 1125 is optionally operated by a different entity than VideoServer System 1120. For example, Game Server 1125 may be operated by thepublisher of a multiplayer game. In this example, Video Server System1120 is optionally viewed as a client by Game Server 1125 and optionallyconfigured to appear from the point of view of Game Server 1125 to be aprior art client executing a prior art game engine. Communicationbetween Video Server System 1120 and Game Server 1125 optionally occursvia Network 1115. As such, Game Server 1125 can be a prior artmultiplayer game server that sends game state information to multipleclients, one of which is game server system 1120. Video Server System1120 may be configured to communicate with multiple instances of GameServer 1125 at the same time. For example, Video Server System 1120 canbe configured to provide a plurality of different video games todifferent users. Each of these different video games may be supported bya different Game Server 1125 and/or published by different entities. Insome embodiments, several geographically distributed instances of VideoServer System 1120 are configured to provide game video to a pluralityof different users. Each of these instances of Video Server System 1120may be in communication with the same instance of Game Server 1125.Communication between Video Server System 1120 and one or more GameServer 1125 optionally occurs via a dedicated communication channel. Forexample, Video Server System 1120 may be connected to Game Server 1125via a high bandwidth channel that is dedicated to communication betweenthese two systems.

Video Server System 1120 comprises at least a Video Source 1130, an I/ODevice 1145, a Processor 1150, and non-transitory Storage 1155. VideoServer System 1120 may include one computing device or be distributedamong a plurality of computing devices. These computing devices areoptionally connected via a communications system such as a local areanetwork.

Video Source 1130 is configured to provide a video stream, e.g.,streaming video or a series of video frames that form a moving picture.In some embodiments, Video Source 1130 includes a video game engine andrendering logic. The video game engine is configured to receive gamecommands from a player and to maintain a copy of the state of the videogame based on the received commands. This game state includes theposition of objects in a game environment, as well as typically a pointof view. The game state may also include properties, images, colorsand/or textures of objects. The game state is typically maintained basedon game rules, as well as game commands such as move, turn, attack, setfocus to, interact, use, and/or the like. Part of the game engine isoptionally disposed within Game Server 1125. Game Server 1125 maymaintain a copy of the state of the game based on game commands receivedfrom multiple players using geographically disperse clients. In thesecases, the game state is provided by Game Server 1125 to Video Source1130, wherein a copy of the game state is stored and rendering isperformed. Game Server 1125 may receive game commands directly fromClients 1110 via Network 1115, and/or may receive game commands viaVideo Server System 1120.

Video Source 1130 typically includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such asStorage 1155. This rendering logic is configured to create video framesof the video stream based on the game state. All or part of therendering logic is optionally disposed within a graphics processing unit(GPU). Rendering logic typically includes processing stages configuredfor determining the three-dimensional spatial relationships betweenobjects and/or for applying appropriate textures, etc., based on thegame state and viewpoint. The rendering logic produces raw video that isthen usually encoded prior to communication to Clients 1110. Forexample, the raw video may be encoded according to an Adobe Flash®standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith,MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, or the like. Theencoding process produces a video stream that is optionally packaged fordelivery to a decoder on a remote device. The video stream ischaracterized by a frame size and a frame rate. Typical frame sizesinclude 800×600, 1280×720 (e.g., 720p), 1024×768, although any otherframe sizes may be used. The frame rate is the number of video framesper second. A video stream may include different types of video frames.For example, the H.264 standard includes a “P” frame and a “I” frame.I-frames include information to refresh all macro blocks/pixels on adisplay device, while P-frames include information to refresh a subsetthereof. P-frames are typically smaller in data size than are I-frames.As used herein the term “frame size” is meant to refer to a number ofpixels within a frame. The term “frame data size” is used to refer to anumber of bytes required to store the frame.

In alternative embodiments Video Source 1130 includes a video recordingdevice such as a camera. This camera may be used to generate delayed orlive video that can be included in the video stream of a computer game.The resulting video stream, optionally includes both rendered images andimages recorded using a still or video camera. Video Source 1130 mayalso include storage devices configured to store previously recordedvideo to be included in a video stream. Video Source 1130 may alsoinclude motion or positioning sensing devices configured to detectmotion or position of an object, e.g., person, and logic configured todetermine a game state or produce video-based on the detected motionand/or position.

Video Source 1130 is optionally configured to provide overlaysconfigured to be placed on other video. For example, these overlays mayinclude a command interface, log in instructions, messages to a gameplayer, images of other game players, video feeds of other game players(e.g., webcam video). In embodiments of Client 1110A including a touchscreen interface or a gaze detection interface, the overlay may includea virtual keyboard, joystick, touch pad, and/or the like. In one exampleof an overlay a player's voice is overlaid on an audio stream. VideoSource 1130 optionally further includes one or more audio sources.

In embodiments wherein Video Server System 1120 is configured tomaintain the game state based on input from more than one player, eachplayer may have a different point of view comprising a position anddirection of view. Video Source 1130 is optionally configured to providea separate video stream for each player based on their point of view.Further, Video Source 1130 may be configured to provide a differentframe size, frame data size, and/or encoding to each of Client 1110.Video Source 1130 is optionally configured to provide 3-D video.

I/O Device 1145 is configured for Video Server System 1120 to sendand/or receive information such as video, commands, requests forinformation, a game state, gaze information, device motion, devicelocation, user motion, client identities, player identities, gamecommands, security information, audio, and/or the like. I/O Device 1145typically includes communication hardware such as a network card ormodem. I/O Device 1145 is configured to communicate with Game Server1125, Network 1115, and/or Clients 1110.

Processor 1150 is configured to execute logic, e.g. software, includedwithin the various components of Video Server System 1120 discussedherein. For example, Processor 1150 may be programmed with softwareinstructions in order to perform the functions of Video Source 1130,Game Server 1125, and/or a Client Qualifier 1160. Video Server System1120 optionally includes more than one instance of Processor 1150.Processor 1150 may also be programmed with software instructions inorder to execute commands received by Video Server System 1120, or tocoordinate the operation of the various elements of Game System 1100discussed herein. Processor 1150 may include one or more hardwaredevice. Processor 1150 is an electronic processor.

Storage 1155 includes non-transitory analog and/or digital storagedevices. For example, Storage 1155 may include an analog storage deviceconfigured to store video frames. Storage 1155 may include a computerreadable digital storage, e.g. a hard drive, an optical drive, or solidstate storage. Storage 1115 is configured (e.g. by way of an appropriatedata structure or file system) to store video frames, artificial frames,a video stream including both video frames and artificial frames, audioframe, an audio stream, and/or the like. Storage 1155 is optionallydistributed among a plurality of devices. In some embodiments, Storage1155 is configured to store the software components of Video Source 1130discussed elsewhere herein. These components may be stored in a formatready to be provisioned when needed.

Video Server System 1120 optionally further comprises Client Qualifier1160. Client Qualifier 1160 is configured for remotely determining thecapabilities of a client, such as Clients 1110A or 1110B. Thesecapabilities can include both the capabilities of Client 1110A itself aswell as the capabilities of one or more communication channels betweenClient 1110A and Video Server System 1120. For example, Client Qualifier1160 may be configured to test a communication channel through Network1115.

Client Qualifier 1160 can determine (e.g., discover) the capabilities ofClient 1110A manually or automatically. Manual determination includescommunicating with a user of Client 1110A and asking the user to providecapabilities. For example, in some embodiments, Client Qualifier 1160 isconfigured to display images, text, and/or the like within a browser ofClient 1110A. In one embodiment, Client 1110A is an HMD that includes abrowser. In another embodiment, client 1110A is a game console having abrowser, which may be displayed on the HMD. The displayed objectsrequest that the user enter information such as operating system,processor, video decoder type, type of network connection, displayresolution, etc. of Client 1110A. The information entered by the user iscommunicated back to Client Qualifier 1160.

Automatic determination may occur, for example, by execution of an agenton Client 1110A and/or by sending test video to Client 1110A. The agentmay comprise computing instructions, such as java script, embedded in aweb page or installed as an add-on. The agent is optionally provided byClient Qualifier 1160. In various embodiments, the agent can find outprocessing power of Client 1110A, decoding and display capabilities ofClient 1110A, lag time reliability and bandwidth of communicationchannels between Client 1110A and Video Server System 1120, a displaytype of Client 1110A, firewalls present on Client 1110A, hardware ofClient 1110A, software executing on Client 1110A, registry entrieswithin Client 1110A, and/or the like.

Client Qualifier 1160 includes hardware, firmware, and/or softwarestored on a computer readable medium. Client Qualifier 1160 isoptionally disposed on a computing device separate from one or moreother elements of Video Server System 1120. For example, in someembodiments, Client Qualifier 1160 is configured to determine thecharacteristics of communication channels between Clients 1110 and morethan one instance of Video Server System 1120. In these embodiments theinformation discovered by Client Qualifier can be used to determinewhich instance of Video Server System 1120 is best suited for deliveryof streaming video to one of Clients 1110.

Embodiments of the present invention may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Severalembodiments of the present invention can also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a wire-based or wirelessnetwork.

With the above embodiments in mind, it should be understood that anumber of embodiments of the present invention can employ variouscomputer-implemented operations involving data stored in computersystems. These operations are those requiring physical manipulation ofphysical quantities. Any of the operations described herein that formpart of various embodiments of the present invention are useful machineoperations. Several embodiments of the present invention also relates toa device or an apparatus for performing these operations. The apparatuscan be specially constructed for the required purpose, or the apparatuscan be a general-purpose computer selectively activated or configured bya computer program stored in the computer. In particular, variousgeneral-purpose machines can be used with computer programs written inaccordance with the teachings herein, or it may be more convenient toconstruct a more specialized apparatus to perform the requiredoperations.

Various embodiments of the present invention can also be embodied ascomputer readable code on a computer readable medium. The computerreadable medium is any data storage device that can store data, whichcan be thereafter be read by a computer system. Examples of the computerreadable medium include hard drives, network attached storage (NAS),read-only memory (ROM), random-access memory, compact disc-ROMs(CD-ROMs), CD-recordables (CD-Rs), CD-rewritables (RWs), magnetic tapesand other optical and non-optical data storage devices. The computerreadable medium can include computer readable tangible mediumdistributed over a network-coupled computer system so that the computerreadable code is stored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the variousembodiments of the present invention is not to be limited to the detailsgiven herein, but may be modified within the scope and equivalents ofthe appended claims.

The invention claimed is:
 1. A method for warning a user of a headmounted display (HMD), the method comprising: capturing images of aphysical environment in which the user wearing the HMD is present, usingone or more sensors disposed on the HMD; processing the images togenerate a three-dimensional map of the physical environment, thethree-dimensional map identifying depth information of real-worldobjects identified in an area of the physical environment; determining alocation of the HMD by tracking movement of the user wearing the HMD inthe physical environment as the user is interacting with contentpresented on a display portion of the HMD, wherein the tracking includescomputing relative distance between the HMD worn by the user and one ormore of the real-world objects in the physical environment using depthinformation captured in the three-dimensional map and from inputsobtained from inertial sensors disposed in the HMD; and generating awarning to the user of the HMD upon detecting the user approaching anedge of the area of the physical environment, based on the tracking ofthe movement of the user, wherein operations of the method are performedby a processor of the HMD.
 2. The method of claim 1, wherein the one ormore sensors are automatically activated upon detecting the user wearingthe HMD or at any time a change in data related to the HMD is detectedduring use of the HMD, or at any time a change is detected in thephysical environment during use of the HMD by the user, the activationcausing the one or more sensors to capture the images of the physicalenvironment in which the user wearing the HMD is present.
 3. The methodof claim 1, wherein tracking movement of the user is done for a time theuser is wearing the HMD.
 4. The method of claim 1, wherein the one ormore sensors include one or more of a depth sensor, or an image sensor,or a stereo three-dimensional camera, or an infrared camera, or anultraviolet camera, or a video camera, or a radar device, or a laserradar device, or a depth camera, or an inertial sensor, or combinationsof any two or more thereof.
 5. The method of claim 1, wherein the depthmap is automatically calibrated to capture the depth of the one or morereal-world objects disposed in the physical environment, the calibrationbased on amount of space available in the area of the physicalenvironment captured in the images.
 6. The method of claim 1, whereinprocessing the images includes combining the images obtained from theone or more sensors and using the combined images to generate thethree-dimensional map.
 7. The method of claim 6, wherein the images areprocessed to distinctly identify the real-world objects in the physicalenvironment, wherein the real-world objects identified includeattributes of a room, a building, a person, a pet, a vehicle, a scenery,or any combinations thereof.
 8. The method of claim 1, wherein trackingthe movement of the user includes, generating a virtual skeleton of theuser within the area of the physical environment; tracking coordinatesof different body parts represented in the virtual skeleton of the user,of the HMD, and of the real-world objects disposed within the area; andcomputing distance of the different body parts of the user, the HMD fromthe real-world objects disposed in the area and from the edge of thephysical environment, using the corresponding coordinates.
 9. The methodof claim 1, wherein the warning is provided as an image highlighting theedge of the area of the physical environment.
 10. The method of claim 9,wherein the image highlighting the edge of the area is blended with thecontent rendered on the display portion of the HMD.
 11. The method ofclaim 1, wherein the one or more sensors are distributed on a front faceof the HMD and are facing outward.
 12. The method of claim 1, whereinthe warning includes one of an audio message, or a visible message, oractivating light emitting diodes of the HMD, or a gradual fading ofcontent displayed on the HMD, or blending images from the physical worldwith the content, or vibrating of the HMD, or disengaging the HMD, ordisengaging one or more parts of the HMD used to render the content, ormoving the display portion of the HMD away from at least one eye of theuser, or any combinations thereof.
 13. A method for warning a user of ahead mounted display (HMD), the method comprising: generating a signalto activate one or more sensors disposed on the HMD, the activationcausing the one or more sensors to capture images of a physicalenvironment in which the user wearing the HMD is present; tracking alocation of the HMD within the physical environment, based at least inpart on the images as the user moves around the physical environment,the location determined by generating a three-dimensional map of thephysical environment that includes depth information of real-worldobjects present in the physical environment and computing a relativedistance between the HMD worn by the user and one or more of thereal-world objects; and sending data for rendering on a display portionof the HMD, the data providing information regarding proximity of theuser to the one or more real-world objects in the physical environment,based on the tracking of the movement of the user, wherein operations ofthe method are performed by a processor of the HMD.
 14. The method ofclaim 13, wherein tracking the location of the HMD includes, computingrelative distance between the HMD worn by the user and the one or moreof the real-world objects in the physical environment, using depthinformation captured in the three-dimensional map and from inputsobtained from inertial sensors disposed in the HMD.
 15. The method ofclaim 13, wherein the data includes descriptive context of a safetycondition detected in the physical environment in the vicinity of theuser wearing the HMD.
 16. The method of claim 15, wherein the dataincludes instructions to guide the user away from the one or morereal-world objects in the physical environment.
 17. The method of claim15, wherein the data includes a warning identifying a severity level ofthe safety condition in the vicinity of the user within the physicalenvironment.
 18. The method of claim 17, wherein when the severity levelis low, the warning is presented alongside any content rendered on thedisplay portion of the HMD.
 19. The method of claim 17, wherein when theseverity level is high, the data includes a signal to render the warningon the display portion of the HMD, the data including a signal to pauseany content that is currently rendering on the display portion.